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FOREWORD 



Tlie Computer Based Instructional Systems Team* of the US Army Research" *T 
Institute for the Behavioral' and Social Sciences (ARI) performs research and 
development in the area of educational technology that applies to military train- 
ing* Of interest are methods for training individuals to develop and utilize 
instructional courseware in reasonable time, at an acceptable cost. 

♦ ^This Techni6al Report describes the development and formative evaluation 
of eigffte*^ Job Aids designed to support users of the Instructional Systems 
Development Model f (ISD, TRADOC Pamphlet 350-30) . The ISD Model is a step-by-step 
procedure for the analysis* dfesign, developipent , itap lamentation, and control of 
military course materials'. To accomplish this research, ARI f s resources were 
augmented by contract DAHC19-78-C-0100 with the Hum^ti Resources Research Organi- 
zation. ' 

Personnel at the US Army Engineer School, Fo^t Belvoir, Virginia, provided 

guidance during the developmental phase of the project: Dr. Everett Rompf,-* " 
Mr . Jack Ainsworth and Ms^ Lois Passman. In -addition, the following individuals 
at the US Army Signal Center and Fort Gordon gave v^uable assistance during the 
needs assessment and evaluation phases: Dr. William Ketner, Mr. Earl Duncan, 
-Mr.— ^Richard Arnold, Mr. John Meyers, Ms. Doris Lakeman, Mr. Dan Danilovich, 
Hs. Yvonne Waldin and Mr. Richard Eberl. Ms. Jean Farrell and Mr. Michael 
Hillelsohn of HumRRO also* contributed to this effort. 



* The entire research project is responsive to the requirements of Army 
Project 2Q263743A794, FY 80 Work Program. 
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"PREFACE 

* 5, • c 

This document reports on a research project in which 18 Job Aids were developed 
to permit implementation of all phases (analyze, design, develop; implement, and control) * 
of the Instructional Systems Development (iSD) model. 7 * ^ 

' ^he research effort was performed, in HumRRO VEducational and Training Systems 
Kvision, Dr. Rpbert J. Seidel, Director. Mr. Russel E. Schulz was the Project Director. 

This research was supported by the U.S. Army Research Institute for the Behavioral 
and Social Sciences under Contract No. DAHC19-78-C-0010, "Computer Interactive 
Author Aiding." The project was monitored technically by Dr. Melissa Berkowitz, Dr. Bruce 
Knerr, Dr. Harold F. OWeil, Jr., and Dr. Leon Nawrocki of theXf.S. Army Research Insti- 
tute. Their support and assistance in this research effort is gratefully appreciated. The 
authors also wish to express gratitude to all personnel at the U.S. Army Engineer School, 
Fort Beivdir, Virginia and the U.S. Army Signal Cento and Fort Gordon, Fqrt.Goidon^ 
Georgia who provided assistance in the research project. Special appreciation is extended * ' 
to the following individuals who contributed to the success of the project: 

U.S. Army Engineer School 

Dr. Everett, Rompf,~Education "Specialist, Individual Training Division * - 

- Mr. Jack Ainsworth, Education Specialist, Individual Training Division 
Ms. Lois Passman, Education Specialist, Individual Training Division 

U.S. Army Signal Center and Fort Gordon 

— Dr. William D. rKetaer, Deputy Assistant Commandant for Educational Technology 
Mr, Earl Duncan, .Education Specialist, Educational Technology 'Division 

Mr. Richard Arnold, fkiucation Specialist, Educational Technology Division 
Mr. John Meyers, Former Chief $ CE Operations Branchy TraiitfrigrAnalyais Division 
Ms. Doris D. Lakeman, Chief, CE Operations Branch, Training Analysis Division 
Mr. Dan Danilovich, Chief, CE Maintenance Branch, Design and Development Division 
Ms. Yvonne S. Walden, Education Specialist, Design and Development Division 
Mr. Richard Eberl, Training Specialist, 6 Design and Development Division " 
• • ' ' c ' 

The assistance provided by Ms. Jean Farrell, formerly of HumRRO, who co-authored 
many of the Job Aids is gratefully acknowledged as is the work of Mr. Michael Hillelsohn 
who authored the Job Aid for ISD V.3 Revise System. Finally, a special thanks to 
Mrs. Alice. Thompson whose excellent work contributed to the design and overall 
appearance of the Job Aids. 
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DEVELOPMENT OF JOB AIDS FOR INSTRUCTIONAL SYSTEMS DEVELOPMENT 



BRIEF ■ ; ; ' 

Requirement: _ i 

The purpose pf the research was to develop and evaluate Job Aids (%ow^ to 
do it" guidance) for the activities identified in the Instructional Systems 
-Development Model (ISD, TRADOC Pamphler^50-30) . The usefulness of .the ISD Model 
depends upon Job Aids which will enable training personnel to translate the ISD 
procedures into instructional products. ^ . 

Procedure: 

A series of Job Aids was designed and developed for each of the five phases 
-o x f -the ISD Model based on needs assessments conducted at the- US Army Engineer "* 
. School (USAES),_ Ft Belvbir, VA and the US Army Signal Center and Ft Gordon • 
(USAf>C&FG), Ft Gordon, GA. Baseli ne da ta were collected from USAES and USASC&FG 
personnel as they performed their~ISD functions without the assistance of the 
Job Aids. /The Job Aids were then provided to individuals who were engaged in 
the activity covered by the Job Aid. Data were collected relating to problems 
alleviated by using ttje 4ob Aid, problems encountered in using the Job Aid, and 
user acceptance of the Job Aid. — — * 

Findings : 

Comparisons were made with the baseline data to assess the value of the 
.Job Aids us used in the ISD process* The findings strongly supported the use- 
fulness of the Job Aids* ^ 

Utilization of Findings: i? 



A summative evaluation, of the Job Aids is plantied to' assess' their value to 
training personnel in the preparation and delivery of a complete course of 
instruction* « 
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INTBODOCTJOIST * " . , / • /• 

The purpose of this section is to provide a brief description of research and develop- 
ment in this area, a discussion of the rationale for this research, and an overview of the 
project history. *> - * • ~ 

A SYSTEMS APPROACH TO TRAINING* ~ '• . ' V 

• ■ * * - • - ' - i * . y 

The systems approach defines a process* which fdcuses'on the job' that is ultimately • % 
jto be performed and upon the individual who,is to learn to perform that job. The systems 
approach is just what the name implies: a* systematic process fo?specifying the desired 
products of training and selecting what; will be taught, how it will be taught, what the 
presentation mechanism will be, and evaluating the effects of each phase of the process. 
It focuses on student^erfprmance as a determinant of content. Its proper application* can 
hardly fail to improve instruction where only incidental attention has been given US these 
functions* However, after more than two decades of research and development In tfris * 
area, the full benefits o| applying the systems approach to instruction Jiiave yetto*be \ . 
realized. . * , : . % i . • ' " « . - 

Early efforts to standardize a definitive technology in this area included those by 
HumRRO (Crawford, 1962; Finan, 1962; Vallance & Crawford, 1962) on behalf of the 
U.S. Army in the early 1960s ang the development 6f the U.S. Continental 'Army*Cpm- 0 
mand Regulation 360-100-1, Systems '-Engineering of Training: (1969) and the U.S. Air ° 
Force Manual 50-2, Instructional Systems Development . (1970). . * - * * \ 

These attempts have culminated in the Instructional v Systems Development (ISD) 
Model (TRADOC Pam 350-30, 1975). This model was prepared by the Center for 
Educational Technology at Florida State University lender contract with the Interservice 
Committee "for instructional Systems and Development, involving the Navy, Air Force, 
Marine Corps, and the-Army. The ISD contains standardized rationale, terntfnology, and 
basic /concepts of instructional systems, ♦* . A ' # 

Difficulties arose in applying th£ ISD approach to military training. Much of the 
early research in^attempting to overcome these problems involved attention to the tech- 
nical difficulties. One of the primary difficulties of the ISD model and its manuals is 
that it emphasizes "what to do," riot "how to dd it.** ISD manuals are intended to have 
general applicability. However, the same methodologies cannot be'applied to the universe 
of training problems. "How to do it" ratify vary widely, even though the specific processes' 
or "what to do" may remain fairly constant across training problems. Initially, the "what 
to do" nature of the ISD manuals was not a severe problem inasmuch as early users t>f 
> the manuals were "expert" training develop. However, later, the possibility of having 
laypersons use the ISD approach to achieve the successes of the experts was explored. 
The use of an ISD manual by military personnel with little or no experience in training * 
program design may cost a fraction of the, cost of hiring or contracting out to experts to 
do the development. However/specific guidance, tools, and procedures are needed by*~ ? 
such Jaypersons for. ISD to be appropriately and cost-effectively implemented within the 
military trainipg establishment. 



THE NEED FOR JOB AIDS ' 

% .In this report "how to do'it^guida/i ifie/h^ls/procedures will be referred to as Job 
Aids. These are; designed to help- a5 members of an instructional staff charged witfi meet- 
ing the objectives, of one or moire comjxments or th§ instructional systems development 
process. t .o 

Job Aids* Jure heeded in all Phases of the ISD process if significant improvements are 
to occur in applying this systematic approach toward producing instruction This approach 
is expensive,, time consuming, critical, and requires specialized capabilities. . 

A wide variety, cf aids do exist, but the majority are more of the "what to do" type 
.than the "hoy t& dcrit" variety. Rather than prividingMictual help in performing the 
work, or even providing detailed how-to-do4t guidance, mostjrf the existing aids may serve 
to reinforce or broaden the guidance provided in the ISD manuals. Aagard and Braby 
(1976) note the practical limitations in using their algorithms and guidelines for general 
purposes; "The task categories and related guidelines are not at a level that will accommo- 
date any training setting/* Briggs' handbook (1970) may also serve such, a broad guidance 

function.-— - 4 ~* . • ■ 

Care should be exertised'in the selection of aids to J>e integrated with* ISD, however, 
' so. that authors are not confused by different yet similar models, sets of jargon, pro^e- % 
' dures* of forms. Existing general manuals ^.differ tifom one another in that they: 

• Include different steps or different jiames for steps. « 

• Include different methods of accomplishing each step. 

• Provide; different levels of specificity, in the details included under each 
step. " * 

?. Provide different formats for reporting the work accomplished unde v 4 
> each step. ^ 

Thus, to provide the most efficient aids to authors, the guidance found in some of these 
* manuals and guide}x>6k$ needs to, be translated and integrated into the ISD-frdmework, . 
rather than referred to in its source form. , * ^ 

. . .Recently, increasing emphasis has been placed on the -development of "how to do * 
it" aidsc A course which guides trainees ^specifying learning activities and in developing 
testy and instructional materials is reported fcjrCNeai and O'Neal (1970). Dyer, Matthews, 
Wright and Yudowitch (1976) prepared guidance for instructional technologists who are 
taskedjto develop or administer questionnaires as part of Army field tests and evaluations. 
In an^ffort to assure the quality of instructional envelopment Ellis, Wulfeck, and 
* Fredericks (1979) developed the Instructional Quality Inventory. Conoley and O'Neil 
(1979) presented a detailed and pragmatic set of guidelines for multi^choice item 
generation and xeview. 

. In the project described in this document, ieb Aids- were developed to provide the 
specific guidance ("how to do it") needed for implementing the ISD model. . , 



PROJECT HISTORY 



On-line Job Aids developed in a previous AR1 project (Schulz, Hibbits, Wagner & 
Seidel, 1979) demonstrated the utility of assisting authors to implement the ISD process. 
^The vokpt characteristics which accounted for the usefulness of the Job Aids are: 

• The Aids jtre query -based and interactive. 

• They are based upon -a realistic authoring needs assessment at an operational , 

" site. , v 

• The Job Aids were developed and evaluated according to ISD guidance and 
recommended procedures. \ \ 

. • Detailed authoring flowcharts accompanied on-line Job Aids. 

0 • The Job Aids were designed Id be system independent, * , ■ * 

• The Job Aids concentrate on assisting users to deal with structural and adminis- 
trative details permitting them to concentrate their efforts on the subject 
matter content of the materials. \ 

The. research project described in this report is an extension of the previous ARI 
project. In the present research, Job Aids were developed to assist Army instructional 
systems development personnel in the analysis, design, development, implementation and 
control of instructional materials. - 

The research began in November 1977. During the research period Job Aids were 
developed and forniatively evaluated.- The ISD model was used to define the general 
activities to be covered in the Job Aids. (Figure 1 shows the ISD model.) Each Job Aid 
consists of two volumes— Descriptive Authoring Flowcharts and a Job Aid Manual!. The 
. latter contains prescriptive descriptions of the activities to be performed, 'along with 
critical references. They wire designed to be stand-alone, step-by ^step procedural guides, 
. equally ^useful to individuals at all experience levels of the Instructional Systems Develop- 
ment process.. The Job Aids were based on an assessment of user needs at the U.S. Array 
Engineer School (USAES), Fort Belvou% Virginia, and the U.S. Army Signal Ceoter and . 
Fort Gordon (USASC&FG), Fort Gordon, Georgia* They were implemented and -evaluated 
afcVhe USASC&FG. - - * " . 

Initially, 13 Job Aids were developed and evaluated, one for each block of the ISD 
model except for Block 1.1, Analyze Job; A Job Aid was not developed for this block 
for the following reasons: 

(1) The research plan called for the formative evaluation of 'the Job Jdds at the 
USAES. Discussions with USAES personnel indicated that the majority of jobs at the ' 
USAES had undergone job analysis. Since personnel at USAES would not be performing 
this activity during the research period, one could not evaluate a Job Aid developed A>r 
this activity. ' . * ' « J 

(2) Job analysis is a highly complex activity that requires input from many sources. 
Many of these sources ue>utside of the USAES, Therefore, it would have been difficult 
to 'obtain the input in £ form needed for the research effort. 9 * 

. (3) Because of the complexity of performingpb analysis, and the multiplicity of . 
Job Aids needed for doing jK>,devek>pment of an adequate Job Aid for this block would/ 4 
require too much of the total research resources. 4 \ 4 
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Figure 1. THE ISD BLOCKS IN EACH OF THE FIVE ISD PHASES 










f 

* 






4 


.13 












feERJC^ 


to r 








i 















' / 
/ 



¥ 



\ 



The evaluation plan for the initial 13 Job Aids was to have USAES instructional 
development personnel use the Job Aids in an actual job setting as they were performing 

_the activities required by the ISD model!^ However, shortly after beginning work it was 
discovered that ail Phase I ISD activities for courses at USAES had been completed. 
Previous information indicated that all ISD Phase I, II, and III activities were bejng con- 
tinuously performed at USAES. Because no new courses were to be established'during 
the research period it would not be possible to conduct a formative evaluation of the Job 
Aids for Blocks 1.2 (Select Tasks/Functions), 1.3 (Construct Job Performance Measures), 
1.4 (Analyze Existing Courses), and 1.5 (Select Instructional Setting) at USAES. In order 
to accomplish the work, a revision in the research 'plan was made to change the location 
at which the aids were to be evaluated. The U.S. Army Signal Center and Fort Gordon 
(USASC&FG), Fort Gordon, Georgia was selected as a substitute for USAES for the 
following reasons: •> 

/ (1) Activities in all phases of the ISD model were being actively worked on at 

USASC&FG. * t 

. (2) The availability of more complete data at USASC&FG would permit more 

-teliable-ami-useful-Jok Aids to result from the research effort. ■ * . 



i (3) Due to the considerable number of personnel available at USASC&FG who ' 
were engaged in the. ISD process, it would be feasible to collect the data required for * 
the formative evaluation in a timely manner requring a minimal delay m project activities: 

Upon completion of the development and formative evaluation of the Job Aids for 
the first three phases of the ISD model another project effort was initiated. It involved 
the development and evaluation of a prototype system-independert. Programming Design 
Guide for an interactive Job Aid. Thehresearch was performed to evaluate the feasibility 
of translating the Job Aids into a computer-based instruction format. 

The Programming Design Guide provided all of the guidance necessary for a user to 
implement the Job Aid on any of a large number of computer systems. The specific 
Job Aid selected for the effort was the Job Aid for ISD Block 1.5, Select Instructional 
Setting. This Job Aid was selected because it was representative of all of the Job Aids 
in fha fire* +hT«* > jtw*** nf model in terms of level of difficulty of translation- 



into computer-based instruction format. 

The Programming Design Guide format used led to a successful implementation of 
the Job Aid for selecting instructional settings. As this Job Aid is representative of the 
other Job Aids, it is expected that the approach used in the development of the Pro- 
gramming Design Guide will be equally successful for Implementing other Job Aids 
on-line. The Programming Design Guide research is described in a report by Schulz, 
UriderhiU, Hargan, and Wagner (1980). 

The successful use of the 13 Job Aids developed earlier in this project for JSD 
Phases I, II, and III, indicated that the interactive job aid approach would also meet 
the requirements for Phases. IV and V of the ISD model. Therefore, five additional 
Job Aids covering the activities in Phases IV and V were developed. Implementation 
and evaluation of these Job Aids occurred at the USASC&FG. 

. The next section of this report provides a detailed description of the analyses 
selection, development, and formative evaluation of the 18 Job Aids developed during 
the research effort. 
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RESEARCH ACTIVITIES 

• * ^ * • 

In this section the research activities involved in the analysis, selection, development, 
and formative evaluation of <the 18 Job Aids will be discussed. Where the research activ- 
ities differed between the development of the first 13 Job Aids (Phases I, II, and III of 
the ISD model) and the last 5 Job Aids (Phases IV and V) the research activities will be 
described separately. (The first 13 Job Aids Will hereafter be referred to as Phase I, II 
and III Job Aids. Similarly the 5 Job Aids for Phases IV and V will hereafter be referred 
to as Phase IV and V Job Aids.) The development of the Programming Design Guide for 
on-line implementation of the ISD 1.5 Select Instructional Setting Job Aid will not be 
disdussed. The reader is referred to the report-devoted to this Guide (Schulz, Underbill, 
Hargan, and Wagner, 1980). . 

The research effort was divided into three major tasks: 

• Task 1: Analysis, and Selection of Job Aids 

• Task 2: Development of Job Aids 

• Task 3: Evaluation of Job Aids 
The activities involved in each of the three tasks are discussed below. 

TASK 1: ANALYSIS AND SELECTION OF JOB AIDS ' 

Task 1 activities were conducted in four steps. These steps are (1) perform Job Aids 
needs assessment, (2) analyze existing job aids, (3) identify candidate Job Aids, and (4) 
refine, modify or complete development of job aids. 

Step 1. Perform Job Aids Needs Assessment 

As shown in Figure 1, the ISD Model consists of 19 major ISD activities (blocks): 
In the Model, each of these activities is'&panded into sub-element activities. For example, 
ISD Block II.l Develop Objectives has been expanded into the sub-element activities 
shown in Figure 2. 

In Step 1 a detailed Job Aids needs assessment was performed to establish priorities 
among the sub-elements of each ISD Mock for. which Job Aids were to be completed. 
These priorities were needed because time restraints imposed upon the research would 
not permit equal developmental emphasis on all sub-element activities. 

" The needs assessment for the Phase I, II and III Job Aids was performed by means 
of a detailed structured interview administered to 11 course develdpers and 10 training 
analysts at the USAES. A copy of the Needs Assessment Inventory used for the inter- 
views is contained in Appendix A. The interview dealt with topics such as the following: 

• ISD products produced by the individual. 

• The ISD input received by the^individual and problems associated with 
use of this input. 
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, • The ISD output produced by the individual and problems associated with 

its production* 

•» <» 

• ISD training available to the individual and additional training required. 

■ • Availability of existing job aids and when available, the usefulness of the 
job aid, f * * 

• Ideas for needed additional Job Aids. • ~~~ 

The results of ttfe interviews indicated that there was a considerable need for Job 
Aids in all sub-elements of the blocks within the first three ISD phases. Specific needs 
were identified for each of the ISD blocks except for ISD Block II.3 (Describe*Entry 
Bahavior) for which interviewees could not identify any specific sub-elements in need of 
assistance. 

The major needs for Job Aids identified in the structured interviews were incorpo- 
rated into a Needs Assessment Review Form which was then administered to 75 instruc- 
tional development personnel at the USAES and 33 development personnel at the 
USASC&FG. For each need listed on the Review Form the respondents were asked to 
rate whether they "Strongfy Agree," "Agree," "Disagree," "Strongly Disagree," or, 
"Cannot Judge." A copy of the Needs Assessment Review Form is contained in , 
Appendix B. 

Major findings from the Needs Assessment Review Form which shaped the content 
of the Job Aids are as follows: 

(1) Specific guidance is needed for completing all ISD activities. . ^ 

(2) * Examples of the products are needed. 

* (3) Sources of information necessary for implementing the JSD Model (e,g., 
where to obtain "frequency of task performance" data) are needed. j 

— -The Needs Assessment conducted for the Phase IV and V Job Aids was conducted 

only at, the USASC&FG with 27 instructional development personnel who were currently 
working in the ISD Blocks for Phase IV and V*' The specific questions covered tot each 
ISD Block are shown in Appendix C/ The major results of this jieeds assessment are 

as follows: * * 

Iffft IV 1 T yw pl o f w iwt Tn trt niMinnal Mnnflgpmpnt Plan. Aids are n e eded which 

provide mathematical formulas for determining resources needed for imple- 
mentation of the management plan. Guidance is needed for reviewing and 
*» revising the Instructional Management Plan. \- < 

— ISD IY.2 Conduct Instruction. The instruction that instructors receive cur- 
rently for this ISD Mock is adequate and as a consequence, there is no need 
for a Job Aid for the actual conduct of instruction. It was agreed that a 
Job Aid for administration and analysis of student opinions concerning 

instru ction would be helpful. 

— ISD V.l Conduct Internal Evaluation. The USASC&FG currently uses a — 
checklist for conducting internal evaluation. Instructional guidance for 
answering the questions contained on the checklist was considered to be 
needed. 

— ISD V.2 Cbnduct External Evaluation. Respondents expressed the need for 
general guidance in performing an external evaluation. • 

— ISD V.3 Revise System. No respondent was able to identify a dear need for 
a specific Job Aid for this ISD Block. 
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Step 2, Analyze Existing Job Aids 

, In Step 2 ISD literature was searched to identify existing job] aids that would be 
useful for the research effort* Two major sources of ^job aid information .was TRADOC 
Pamphlet 350-30, Interservice Procedures for Instructional Systems Development (1975) 
and the McDonnell Douglas Corporation Bibliography of Aids for' Course Material Devel- 
opment (Logan7l976). -Each job aid identified was v carefully analyzed to determine:. * 



• Its relevancy* ■ — ; ~ ^ 

• How it fits into the sub-element blocks. 

• How it feeds information into other ISD blocks. 

• The degree of similarity with other job aids. 

• How it might be combined with other job aids to form a more compre- 
hensive Job Aid. \ 

' . . • The amount of the total research effort that would be required t^ complete 

the jolraidif it were selected f or-indusion in-the final listing of the jQb_ 

•Aids to be completed. • . V 

In general, no existing job aid was identified that* could easily be translated into the 
interactive format proposed for the Job Aids to be developed in the research effort. How- 
ever, the literature search did produce information that was cataloged for later use in Job 
Aid development. For example, much of the ISD materia? produced at the USASC&FG 
later proved to be extremely valuable in the Job Aid development effort. 

Step 3. Identify Candidate Job Aids - . . \ 

Based on the Job Aid needs assessment conducted in Step 1 and the analysis of 
existing aids performed in Step 2, candidate job aids were identified in Step 3. 

Step 4. Refine, Modify, or Complete Development of Aids * 

It had been anticipated that existing aids wcfuld be identified that would be useful in 
the Job Aid research effort* It was thought that those aids would in most cases require 
only minor mndifinatinn far presentation in an interactive format. This modification 
was to be accomplished in this Step. However, as already noted in the discussion of 
Step 2, no job aids were identified that could be easily modified for presentation in an 
interactive format. As a consequence, this step of the research was not performed. 
R&ther, the information contained within some existing job aids -was incorporated into 
.'the Job Aids developed Jn this project, 1 ° , 

TASK 2: DEVELOPMENT OF JOB AIDS 

Ptertous researcMSchu^ 1979 ) 



indicated the utility of preparing Job Aids that make use of a flowchart format. While the 



Job Aids developed in this project, were intended to be used initially off-line in a printed 
version, they were to be developed so as to facilitate possible later translation into on-line 
computer versions. Therefore, each Job Aid developed in this projept is composed of two 
documents-Descriptive Authoring Flowcharts and Job Aid manuals.' In addition, there is 
a document which provides an introduction to the use of the Job Aids and general infor- 
mation regarding useful resources for ISD implementation. 



ERiC ' .18 



Job Aid Format 

Introduction to the Use of Job Aids and Job Aids Resource Manual, The introduc- 
on describes tiie Job Aids, how they were developed, and provides general information 
6n how to use the Job Aids* The Job Aids Resource Manual provides general information 
concerning ^OBAP (Comprehensive Occupational Data Analysis Prograqas), conducting 
field surveys, 0 and tiienisfe^various i»nek of personnel in. the ISD process; Figure 3 : 
shows the major topes- indudedon the Introduction to the Use of Job Aids and Job Aids 
Resource Manual* 

Descriptive Authoring Flowcharts* ThcfDescriptive Authoring Flowcharts are the 
primary documents used in the Job Aids. They direcTthe-usg^specific guidance, 
examples and references providedlin the Job Aid Manuals. FigiufT^^qwa represent*- 
.tive page of descriptive authoring flowcharts. The flowcharts ,are easy tou&e^n^ha^ey 
include only five flowchart symbols- The symbols are standard flowcharts symboiTSxeep^ 
for the one that looks like an open book or manual. This symbol is used to cue the user 
that he/she is to go to the Job Aid Manual for additional information or guidance. The 
directions contained within the flowchart symbols are frequently supplemented by narra- 
tive informationprov^^ Descriptive Authoring Flowcharts. This 
narrative information is sufficient to.provide an overview of the activities that are to be— 



uwauvv; iwwi iwmw u «, r w,^ — ~ w ~ ■ 

performed in the Job Aid. T*e Descriptive Authoring Flowcharts are also sufficient, to 
act as an executive summary for supervisors or commanders. 

Job Aid Manuals. As stated above, the Job Aid Manuals provide the specific guid- ^ 
ance, examples, and references necessary to help the user produce the products for the 
ISD activities covered by the Job Aid. In addition, each Job Aid Manual has associated 
with it one or more worksheets to use in the development of each product. When the 
user is referred to the Job Aid Manual an attempt has been made to provide four types 
of information: (1) the purpose for performing the activity called for in the-flowchart 
Mock, (2) sources of information for performing the activity, (3) specific step-by-step 
guidance for performing the activity, and (4) if the activity involves filling in a worksheet, 
an example of what the worksheet looks like after ths activity has been performed. In 
addition, the flowchart block for whicb*guidance is given is reproduced at the top of the 
page of the Job Aid Manual.; Figure 5 shows ajrepresentative page. from a Job Aid Manual. 
(Note that flowchart block 9 is also represented in Figure 4.) 

As previously stated, the Job Aids were intended to alleviate the needs exposed 
during the survey. I n some cases, guidance is readily available from 



other sources for meeting the needs; or inji few cases, time constraints placed on the 
research project would %ot permit the development of some needed guidance. As a con- 
sequence, the Job Aids do not always provide complete, guidance or information for all 
activities in. the ISD Hock covered by the Job Aid. Therefore, ja brief summary of the 
Job Aids for each ISD Block is provided below. - 

The Job Aids for the first three Phases of the ISD Model are available as Army. 
Research Institute documents (Schulz & Farrell, 1980a, 1980b, 1980c, 1980d, 1980e, 
1980f). Job Aids for the remaining phases are iii press at this time (Schuk, Wagner & 
Hillelsohn). The Job Aids are grouped by ISD Phase. That js, there is a separate Dwcrip- 
tive Authoring Flowchart manual for the analysis, design, development^ implementation, 
and coptroi phase of the ISD model. Similarly, there is a Job Aid Manual for each of the 
phases! . ' . 
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INTRODUCTION tQ THE USE OF JOB AIDS 


> 


4 [ » ' 

; - "\ 


' What are Job Aids? 

What is the Classification System Used in TRADOC Pamphlet 350-30 and 
Job Aids? ' - , * 
^ What Sources of Information Were Used in the Development of 
Job Aids? 


- : - v~ 




What Job Aids are Presently Available and What Do They Consist of? . 
What are the Main Parte of the Flowchart Manual? V 
What Flowchart Symbols are Used in the Flowchart Manual? 




c 


What Instruption is Provided Within the Flowchart Symbols? 
^-^What is the Purpose of the Supplemental Information Provided in the 
^^EUwchart Manual? * . / 4 
What js the^Hovvchart Block and Page Numbering System? 
How Do 1 Use the^Jab-Ai^ and What are the Main'Parts of it? 
_What,ii the Purpose of thePai^h^Qyvcharts at the Top of Some Pages 

of the Job Aid Manual? ^ 
What is the Purpose of the Questions Written in Scrif>TTfia^ 

Some' Pages of the Jpb Aid Manual? * ~ ^ ^ 






What is the Purpose of the Completed or Partially Completed Examples- 

. of Worksheets? 
What Does This All Mean and What Do 1 Do Now? 

JOB AIDS RESOURCE MANUAL 






How Can the. Resource Manual Help Me? 9 
What Are Some Sources of Job Significant Data? 
Which Sources of Data are Included in the Resource Manual? 


■2. 

t 


*• 


Figure 3. MAJOR TOPICS INCLUDED IN THE INTRODUCTION TO THE 
USE OF JOB AIDS AND JOB AIDS RESOURCE MANUAL 




- \ ... 

. • 

.ERjC . 


• 

.' , -20- . ° " 


- < 

✓ ^ 
1 

« 









9 




Obtain Critical Task 
Idtntification Sheets 
"(CTI) 



No v 




7 , 



• Task inventory include* all tasks performec 
in this skill level. 



Guidanct for Obtaining* 
Task Inventory List 



•- Task invtntory should lit available as output 

from ISO 1.1 Analyze Job. - 
o 

• If procedures for listing tasks by skill level 
' wtra not complete, see persons responsible 

for compiling task list. — ; — 



• VVhen list has been obtained return to Manual, 
ISO 1.2 Select Tasks for Training, Block 6, 
pageA-,11. 



Critical task identification and final selections are recorded on the Critical 
Task Identification Shut (CTI). The CTI Sheet, therefore, prides a , 
documented record for these decisions and the basis for each. . 

For guidance in obtaining CTI Sheets, see Manual, page A- 12. 



Information needed for "setting up" the CTI Sheet includes the following: 

- MOScode 

- Skill level. - 

- Task 10 numbers < 

- Task inventory 

For sources of information see Manual, page A-13. 



• Number each CTI Sheet (Use as many as necessary to list all.the tasks). 

• Attach a copy of the task inventory to the first CTI Sheet tcThelp you 
identify the tasks to which the numbers refer. 

• Example: See Manual, page A-14, 



Figure 4. REPRESENTATIVE PAGE FROM DESCRIPTIVE 
AUTHORINiTFLOWCHARTS 



from block 8 




• ukmm ***** tw "mm »"ti»CT| pm ***** m mmmr 
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to block 10 



TVfat infatMUtfl** cU 1 H€€d and (June d* 9 fct.itf 

• The following information is needed before you can aet up the CTI Sheet. Source's 
for locating each item of information are identified and in most, cases throughout * 
this manual the source will be rated as "excellent/ 1 "good/* or "fair." Under- 
standably not all sources of information will be readily available to you, 



: 

Information Rtquirtd 


. » Source(s) 


Rtrmrks ^ 


MpS and Skill Leva! 
designation 


• Supervisor 

• AR 61M (for a new MOS) 

• Output of ISD lit Analyze Job 


• Excellent source 
* *? Excellent source 

* Excellent source 


Task 1.0. Numbers 


• Numbering system prescribed in ^ 
> TRADOC Circular 351 28, if already 
. assigned ' * \ 


* Excellent source 




• Temporary numbering system of your - 






own design, e.gt 1 through 500, if * 
authorized numbers have not been 
, assigned 


e ^Excellent source ' 


Task inventory/ 


• Output of ISD I.I or job analysis 

• List currently in use 


• Excellent source . 

• Good source 



/ ' . * 

> " Figure 5.- REPRESENTATIVE PAGE FROM JOB AID MANUAL 



PHASE I - ANALYZE ' > 

ISO 1.2 Select T»k$/Function$ 

The Job Aid for this block assists the user in making the decision of which tasks are 
to be selected for training and which are to be rejected. The output from using the Job 

— LTV. • A final list of tasks for training. ~ 

• A list of tasks rejected. 

" • A summary of the data upon which the selection decision was based. 

ISD 1,3' Construct Job Performance Measures • / " 

■ ./ 

This Job Aid provides guidance for performing a task analysis of each of the tasks 
selected for training. The output from use of th^ Job Aid is: 

, • Information concerning the conditions under which each task is performed. 

• Cues indicating when it is performed. 

• • /Standards indicating how well it should be performed. 

• References cand training tips. _ ~~ 

• Identification of task elements or performance steps. 

This information is recorded in a format which can be placed in the Soldier's Manual, 
understood by the soldier learning the task, and used as an evaluation tool by someone 
supervising the task. « 



ISD 1.4 Analyze Existing Courses. ' ■ * . _ \ 

The purpose of this Aid is to "eliminate duplication <A effort in several ISD processes. 
Thus, it contributes to one of tije overall goals of ISD^Ho increase thecooperatfve devel- 
opment and use of training throughout the military services. Specific guidance is provided 
for analyzing existing courses. ( The output from this Job Aid can be: 

_ • The generation of an initial task list for a particular MOS. 

• The identification of suitable courses for training. 

* Thfi idpntifirfltion of courses, or parts of courses, which may be suitable 



after revision by course development persomtek- 

' , * • - / 

ISD 1*5 Select Instructional Setting 'J 
* * *» » • 

,The purpose of the Job Aid for this ISD Block is to provide guidance in choosing 
instructional settings (training locations) for tasks selected for training. Use of the Job 
Aid will result in a listing of &1 critical tasks in whicf^ieach ta^k is assigned for training 
to one of the following instructional settings: V 

• Institution (resident school training). 

• Supervised on-the-job ^twdning (SOJT). . x 
* * 

; • Self-study: ^ 

<s * -■ ■ • 
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PHASE II - DESIGN , 



ISO 11.1 Devtlop Objective 

* i 

The cpurpoee of the Job Aid -lor this ISD Block is twofold* First, it guides the 
user in the actual writing of correctly stated learning objectives. Second, it explains the 
.mechanics of deriving learning objectives from *Task Su&mary Sheets (developed in ISD 
L3 Construct Job Performance Measures) by means of learning analysis. The outputs 
from use of this Aid are: * ° 

• Termiiftd learning objectives tot each task to be trained. 

• Learning objectives for task elements. ^ . 

ISD II.2 Devtlop Tests 

• *; * . 

-In this Job Aid the user is provided guidance in deciding Aether tasks and task 
elements should be tested by means of written or performance type test items. Infor- 
mation and guidance is also provided for the preparation jof written test items. The Job 
Aid does not cover the development of performance tests, nor jfloer it cover preparation 
of administrative materials or validation of test items. The output consists of written 
test items which will be used to test 'the ability of soldiers to correctly perform critical 
tasks or task elements. 

ISD II.3 Describe Entry Behavior 

The goal of the Job Aid for this ISD Block i& to provide the guidance necessary to 
test the entry behavior of trainees and to. utilize this information to adjust the beginning 
point of instructional units; and to verify or revise the assumptions made in the learning 
analysis performed in ISD H.lj Develop Objectives, about entry skills and knowledges to. 
be trained. The output from using this Job Aid is a validated learning analysis for each 
task. 

■ • * * 

ISD 1 1. 4 Determine Sequence and Structure^^ 

This Job Aid provides guidance for sequencing and grouping learning objectives to 
-help-assure that when instructional materials are developed, jeach learning objective is 
placed in the best relationship with other .learning objectiveTto: 

e Produce the most learning in. the shortest period of time. ' 

e Help the soldier make the transition from oneskill or body of knowledge 



4a-another> 



e Assure that supporting (enabling) knowledges and skills are acquired before 
subsequent subject matter is introduced. 

Use of the Job Aid will. result in the proper sequencing of all tasks and task elements. 
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PHASE III - DEVELOP v 
ISD 111.1 . Specify Learning Events/Activities . 

.. me purpose of the Job Aid for this. ISD Block is to help the user to select -appro- 
" priate learning events and activities for training sessions in which critical tasks, will be 
taught. Learning research' has identified learning activities and conditibns whichjre 
essential for teaching.different kinds of learning objectives. In this Job Aid, the user 
learns what those activities and conditions are. Use of this Job Aid-will result in a listing 
of learning activities that could be used for teaching the learning objectives for each driti- 
eel task*. _■ 2 ' - * 

ISD III.2. Specify Instructional Management Plan and Delivery System 

The Job Aid for this ISD Block enables the user to apply a systematic approach to 
the selection of delivery systems' (media) for presenting instruction to trainees. This 
systematic approach to jnedia- selection requires, consideration of the nature of the objec- 
tives: their learning categories/subcategories, learning activities, and instructional settings, 
Such factors as equipment and facility constraints, cost of procurement, cost of replace- 
ment, and development costs are also considered in the Job Aid. However, the Job Aid 
does not provide guidance for conducting a cost-effectiveness analysis of media, nor for 
the actual development of the Instructional Management Plan. The output from this Job 
Aid is a complete identification of the characteristics of a delivery system to be employed 
for! teaching each task and task-el^ment. , ^ 

ISD 111.3 Review/Select Existing Materials 

In the Job Aid developed for this ISD Hock'fuidance is provided to help instruc- 
tional developers evaluate existing materials in order to determine their usefulness in the 
- training program they are preparing. The Job Aid results in, a list of all existing materials 
which have been considered for use in the course developmentactivity. Each item of 
material is evaluated as to whether it should be accepted or rejected for use in the course. 
If the material is to be accepted, suggestions for revisions (if needed) are documented. 
Each item of material is therefore classified as: * 

i * •.Accept Jot use without revision. 

• ' Revise for use. ' 
• Reject* 

ISD III. 4 Develop Instruction 

The Job Aid for this ISD Hoclfhas. two goals. The main goal of the Job Aid is to 
assist m p^oduemgtraining materials for the following media: audio-only, audio-visual, 
and written text. A secondary goal is to familiarize the user with the ways in which .the 
•various production personnel can help in the instructional development effort. The ' 
product resulting from use of the Job Aid is a first draft of instructional materials. 
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5 Validate Instruction 

i# • - ~ v 

Guidance for conducting two types of instructional validation is provided in the Job' 
Aid for this block of the ISD ihodeL *they .are: individual validation trials and group 
yaKdatioivtAals.- Validation as used in this block of the ISD the process by jrtiich ' 
instructional material (course, module; lesson, etc.) untested Andjevised until it teaches 
what it was intended to teach. - The Job Aid defines eachof-these types t>f trials and 
provides guidance in when and how to use them. The outcome' from use of this Job 
Aid is a set of validated instructional- units. 1 % 1 1 ' 

PHASE IV * IMPLEMENT 

ISD IV.1 Implement Instructional Management Plan ■ 

The purpose of the Job Aid developed for thi? ISD Block, is to' provide guidance in 
the techniques to use for reviewing and're vising (when necessary) the Instructional Maftage- ' - * - 
menjt Flan developed in ISD III.2. The output from the use of thjs Job Aid is,«n accurate' 
Instructional Management Plan suitable for implementatiori in' ISD IV.2 Conduct ^ ^ * - 

Instruction. ■ * #t ' * v « * < 4 

: , ^ ■ • : 

ISD IV.2 yConduct Instruction » . \ 

"This Job Aid emphasizes the collection of student performance and survey, data ^ 
during the conduct of instruction. It does not cover the actual conduct of* instruction • ' , \ 
inasmuch as this .guidance is available from a number of other sources. Student Survey 

• Foray and Instructor Survey Forms are provided which permit students/instructors to ' 
provide^input as to needed changes in. the instructional program. The Job Aid provides ' * /> : * 
guidance as to when the oirvey forms should be administered and how to summarize < \, 

' data received fropa ihem. The output from this,Job Aid will be student/instructor' data * 
useful '{or determining needed changes in the instructional program. Student/instructor 
data are addressed in jhe internal evaluation report prepared in ISD Block V.l. ' . 

PHASE V • CONTROL - * , . 

______ « . * 

. _. - . " . " ; " * - \ . - - 

ISD V.1 Conduct Internal Evaluation * 

The purpose of the Jotr Aid for this ISD Bkick isHo provide the user with guidance # * 
in conducting an internal evaluation of courses whidi were developed using the Job Aids 
described above. ' The Job Aid assists the user to answer the following questions: . < 

• "Is the instructibn providing the students with the necessary knowledge 
and skills to meet the objectives in a satisfactory manner? 1 * 

• "To what extent did the development effort conform to the ISD process?" 

' • "How good are the products resulting from the various ISD activities?" 

A detailed checklist and guidance, for its use is provided for gathering the information 
needed to answer the above questions. The output from this Job Aid is an Internal 

\ Evaluation Report wkich is used to modify (.where necessary fthe products from previous 

\ISD Blocks. i 
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ISD V72 Conduct External Evaluation 



\ . ^Thie purpose of the Job jdd Jor thi s TST) Block i s to h e lp th e us e r to eoriduei 

external evaluations of training programs. Emphasis is given in the Job Aid to determine 
whether a school-taught course adequately trains its graduates. Graduate and Supervisor 
Survey. Forms are provided in the Job Aid and instructions are given as. to how these are 
to be administered and how the data derived from the survey are to be summarized. The 
output from use of this Job Aid will be an External Evaluation Regort which will be used 
in the next ISD Block, V.3 Revise System, to determine revisions needed in the instruc- 
tional program, o . > 



ISD V.3 Revise System 



carc/fi 



GOidance is provided in this Job Aid for making a careful analysis of data relating 
to doctrine changes, soldiers' performance on Skill Qualification Tests (SQT), and instruc- 
tional program, deficiencies reported in the Internal and External Evaluation Reports. 
Algorithms are provided for determining required course revisions based on the above 
information. ThrputcOBi^ftom this Job ^id will be a report detailing the revision* 
needed in th£ instru<^6nal^)gram : . %r 

The Job Aids described above were fonnatively evaluated at the USASC&FG. The 
formative evaluation was conducted in Task 3, Evaluation Job Aids. A discussion of 
Ta^k 3 follow?. • . * ^ - 



TASK 3: EVALUATION 



/ 



OF JOB AIDS 




Evaluation of Job Aid/ - Level 1 (Informal Review of Known Aids 

: The first level4valuation,in this project was carried out as part of th^Job Aid devel- 
opment effort, ami is described more fully in Task 1* Project staff reviewed, published 
guidance and,taa&wn iSD aids, attended workshops on ? the use of aids puch as the Instruc- 
tional QuaUty/^yentory (IQI) (Eilis, et ah, 197^9), and visitedVnumlier of XRApQC 
installatio; is Jtp learn about any -useful forms, instruments, or guidance that should be 
incorporate^ into the Job Aids* 

Evaluation of Job Afcte- L#vel 2 (Formative Evaluation 'of New Aids) 

* * * . » * 

»viqus information indicated that all ISD Phase Mil activities were being xontinu- 
u performed at the .USAE^^Ft. JBelvoiir, Va. However, in early visits to USA^S, it 
discovered that it, would not be possible to collect evaluation data for ISD Phase I 
Aids inasmuch as there was no one who would be hoiking in thirarfea during the 
proposed evaluation period. The USASC&FG at Ft: Gordon, Ga., was chosen as the' 
altemative s location, and evaluation plans weremodified to accommodate the new site. . 

In Task 1, those ISD activities in most neea^f procedural guidance and assistance m 
were iden^fied. Ifr was agamst 4toese needs that the Job Aids were evaluated. The focus 
of .this evaluation was whether or not the problems noted during the needs assessment 
were' solved or ameliorated by the Job Aids. > 

* A detailed formative evaluation plan was prepared which set up procedures for 
gathering evaluation data. The evaluation plan was described to USASC&FG personnel, 
and approval obtained prior to its implementation* 
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.One of the features of the evaluation {dan was the collection of baseline data. Its 
purpose was to determine how well school personnel were performing- ISD Phase Mil 
activities without the assistance of the "Job Aids to be provided in this project. A copy 
of the instrument used for the collection of~baseline data is shown in Appendix D* 

Following the collection of baseline data \the Job Aids were provided to Ft* Gorddn 
personnel to use in their ISD activities. USASCJ&FG personhel were given a very short 
training session in the purpose and use, of the' Job Aids* It was emphasized in the train- 
ing session that the focus of the evaluation was the Job Aid itself, qx>t the person who 
was working with it Job Aid deficiencies jarere to be pointed out, but so too were those 
aspects of the Job Aid that were deemed to be helpful (e.g., more efficient procedures . 
or specific guidahce, etc.).^ < 

After-each individual completed using the Job Aids he/she was interviewed concern- 
ing the practicality and usefulness of the Job Aids* The structured interview form used 
for these interviews is shown in Appendix- E. In addition to interviewing Job Aid users, 
their immediate supervisors wtee also interviewed* In this latter interview the major . 
emphasis was the supervisor's perceptions of any reduction or improvement in the 
efficiency^ of their subordinate's work, from the use of the Job Aids. Finally, all person- 
nel (Job Aid users a£n& their immediate supervisors) were asked to identify specific 
changes, deletions, or additions that^woufd increase the accuracy, timeliness, and useful- 
ness, of the Jo b-Aids. Any serious problems t hat were iden tified were attended J6„as i 

T3f^n"as^pdssible and revisions mad e to the Job Aids* * 
\ BTtRe evaluation of Phase IV and V Job Aids r% "expert" review was the* main 
approach ijsed in assessing the Job Aids developed for these ISD Phases* Opinions 
regarding user acceptance were also obtained, as was information on problems that these 
personnel expected to experience while using the Job Aids. ^ ~ 

Evaluation of Job Aids - Level 3 (Impact on Instruction) 

The Job Aids were designed to facilitate many steps in the ISD process J -ahd as such, 
ultimately .should hay* an effect on instructional quality and relevancy. Those, criteria % 
are measured by a change in the behavior of students who undergo training in tfte cWse* 
resulting from the ISD process* Although accumulation of student data could have 
indirectly served to assess the* adequacy of the Job Aids, it was decided that this activity 
not be performed as part of the formative evaluation of the Job Aids. Rather, more 
attention was paid to the determination of Job Aids 9 usefulness and the need for" 
improvements* , , 

' * * * 

Findings * • 

The formative evaluation plan proposed in this project produced much useful 
information for the revision and improvement of the Phase Mil Job Aids. The plan 
was followed for the Phase Mil Job Aids to the extent that^ • 

^Baseline/data were obtained from.USASC&FG persohnel regarding their 

ISD activities before Job Aids were introduced. 
* ** * ~ ' 

. s ' — Acceptance and use data were obtained* following intervention with the Job 
Aids from some of the personnel from whom-baseline data were collected* 



The data obtained by the procedures outlined above were examined to determine 
the acceptance and utility of the Job Aids', and to seeif any improvements needed to be 
niade thaJ^ould increase their efficiency and/or effectiveness. The effects of the Job Aids 
were mo* readily observed when the same USASC&FG personnel who participated in 
Taik i (needs assessment) were the ones' from whom the date were obtained regarding 
their use of the job Aids. Date obtained from others highlighted the generality of the 

J ° b ^a^und'date were collected from personnels U.S. Army Signal Center and Port 
(k>rdon7^ Gordon, Ga., who were engaged in activities covered by ISD Phases I-ffl. Back- 
ed information Forms and ISD Activity Checklists (see Appendix D) were admin- 
istered to over 200 individuals. .Approximately 50 of these individuals were visited one 
week later and interviewed by project staff concerning. their ISD activities during tee 
preceding two-week period. The specific individuals interviewed were selected on the 
basis of the ISD Activity Checklists completed the week before. In tins way, a compre-- 
heVfiveletoTinfonnation was gathered regarding the job activities of personnel workuig 
in the ISD process. The problems they encountered, the time to perforacertain 
activities, and their opinions regarding the effectiveness and efficiency of the ISD 
process (prior to the intervention of Job Aids), were among the topics of most mterest 
teat ware accumulated in the baseline date. Similar sets df date were obtained following 

their-use of-the-Job -Aids . — — — *-- — • j 

The results of this evaluation were unexpected. First, the intensity and extent of . 
needs expressed by the USASC&FG and USAES personnel was much greater ..than had . 
teeTantidpated As was described in Task 1, Job Aids were perceived as needed in all , 
blocks of ISD Phases I-III, and covered practically all activities within each block. These 
needs were supported by the baseline date collected in the ISD activity sheets. These 
datfshTwed teal the ISD activities took considerable time, due in pari ; to the _ inaccurate , 

. and untimely information generated by inexperienced individuals With tins situation - 
as the baseline condition, the reaction to the Job Aids was judged as positive. 

All (100%) of the individuals given the Job Aids thought they were Good or 
Excellent. The only negative comments were editorial in nature. Problems with the . 
ftowchaj^format, as seen by its developers, were not confirmed by the users . Follow-up 
structured mtervie^see AppSTdS TE) with 57 of ^ ^ ehc ^ ^Z^'~^ 
for the draft Job Aids, that a change in project plans seemed essential. For example 
therewasthe refusal by some of the users to part with the draft Job Aids on which their 
edtSnTcominents were written,' unless they could be replaced with other copies. Ako, 
copies of the Job Aids were requested by other personnel in the school who , wer e i net 
airticioants in this study, but wanted to use them in their own ISD work. In addition, 
teTThase I and II Job Aids«are being incorporated into a USASC&FG course that 
teaches the analysis and design of instructional development. This is only some of the 
evidence of acceptance that influenced the change in direction taken in the Project. 

Evaluation plans were curtailed and resources shifted to Job Aid development which 
permitted a broader set of activities to be covered by each Job Aid than had been 

^nthe foStive evaluation of Phase IV and V Job Aids the date emphasized 
editorial changes to the documents. While it is true that some of tee same personnel who 
responded in ti?e needs assessment, evaluated the Phase I-III Aidses was not the case 
' Jor tee Phase IV and V Job Aids. Due to time and resource constraints, the ultimate users 
. were not able to participate in the evaluation. Rather, only "expert" critiques of the 
J2 Aids were okained and the Job Aids modified accordingly However, these Phase 
IVand V Job Aids will be sent to USASC&FG for review prior to their release by ARI 
for general use. 



RECOMMENDATIONS 



■ 



A major shortcoming of the evaluation was the fact that the continuous nature of 
the ISD process was not accommodated. "Thatjs, each Job Aid, was prepared for use as 
a "stand-alone" set of documents that would assist those engaged in the activities covered 
by a given ISD block. It was assumed that the information feeding into that block— 
the. information necessary for using a particular Job AicHwould be correct and available 
in a timely manner. This meant that the output .or products of one Job Aid would have 
to be available f or use by those who needed that product as input to their own Job 
Aid. Thus, when the evaluation of Phases Mil Job Aids was planned to occur simul- 
taneously, an adequate evaluation of the impact or utility of the Job Aid was made 
extremely difficult and impractical within the constraints of this project's time and resources. 
_A»l<me resutt, the pre- and post-use - 

measures of the Job Aids, to a detailed critique and debugging of the Job Aids by actual 
and potential users. There is still the need to evaluate the utility and impact of the Job \ 
Aids when applied to the ISD process as~a whole. 

First, the Job Aids need to be consolidated. There are many examples of redundancy 
and duplication of forms and effort that jhould be eliminated. " 

Second, the Job Aids should be used throughout the entire ISD process for a given 
. course development effort. In this way, a realistic, 'longitudinal evaluation of the Job 
Aids can be accomplished. The contribution of each Job Aid to the ISD process, and the 
eventual impact upon instructional quality and relevancy, could be assessed in this type 
of evaluation. 
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APPENDIX A 

* - • - / 




- NEEDS ASSESSMENT INVENTORY FOR ISD PHASES 1. II, AND III 
• ~~ . • JOB AID DEVELOPMENT / 




• . # • 

1. IDENTIFICATION 


„ » - - 


1. N nmA - 2 / : : : * 

2. Title; 1- = — 


r 


,31 Organization: L '. . : : 

4. Are you a subject-matter expert in any military area(s)? If so, what area(s)? 


t • : 






* * 

• * 5. Have you ever been an instructor at USAES? If so, what courses, positions, 
\* • when, how long, etc.? 




^ - — , 

6- How long have you worked in ISD? / 


- 






II. JOB 




%. In which JJSAES courses do you work or have responsibility for someone 
else's *7ork? ■ / 


m % 
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IL • . 




• - 


2. What are your responsibilities? - 


v v - • 


3, In what ISD activities are you presently working or supervising others? 

How longrfor each? . ' vr 

* ■ * - * ^ 




. " - - : ;3 

• ~ ••- - ' ' f * ' '*2 


* .. ✓ 




o 4 # what additional ISD areas have you worked on (or supervised in) aunng tne * 
past year? How long tor each? 


■! - - . # * 


- • - * ' i ' " - \ * ' 




What did you produce? ~ ~r~ 




* 
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" * * 


. * * * % / • * .* . * . . * \ 










«' - 


HI. INTERFACE ^ \ 




- > 


1. From whom dp you (or the individuals* you supervise) receive input? 

1 " ' t 







.i*—** /** ******* 




^ \ / 


/ • 

/ 4 

1 ' • -< • * 

a. What problems, if any, do ypu (or your staff) have in using the input? 


t 




-■ . o < f i" 0 






• •* 4 












/ • * 
* b. How much time per day /week/month do you (or yourstaff ) spend on 
- / the related tasks because of the problem? 

* » * 
i . , • 






j c. What ~controldo you-haveover the quality of the input? , * , 


f. *' 


v , i 
<# 


d. Do you have any suggestions as to Ijpw this input could foe improved? 

\ - 0 

! . • . ' ■ . » \ *\ 






**• 

- : . ' ' ! • ' s ■ 

*> * 

. ! 








r-mc 






I^TT™""™ -1 ' 


a ' ; . • • , j / . _ . _ 





JU W hat training have you had (mi litary or civilian) on subjects related to 
instructional Osteins developm^l^BducationaHechnolog^ ^d Jflte Uke? 




2, What additional training, if any, would you v like to receive? 



3. What job aids are available to you or your staff for performing ISD activities? 



a. IWhich of these do you actually use? 



- / 



b. How useful are the job aids? 



r 



c. * How could they be improved? 



ERiC ; 
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4. What locally produced forma do you (or your ataff) complete relevant to 
ISD activitiea? . ' > 



Which, if any, of these forms do you/your staff use? (Show interviewee 
appropriate forms shown in ISD manual.) 



6. What happens to the completed form? (How is it used, by whom, etc.?) 



V. OTHER PROBLEMS 



-4* 



1. What other problems do you encounter in your 'work that have.not been 
^^mgnttonedjwrlier? — . ' '__ 



- ERjC 
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2. 8b you have ideas about possible improvements in the ISD process that would 
'help solve this problemQs)? . % 



VI- IDEAS FOR ADDITIONAL AIDS 



1* If you had the choice of selecting five ISD aids for development, what ISD 
ureas would they cover? ~ " 



2, Do you know someone who can provide additional ideas on the subjects we 
have discussed? 




NAME:. 
TITLE:. 



APPENDIX B 

/IEW FORM F( 
JOB AID DEVELOPMENT 



NEEDS ASSESSMENT REVIEW FORM FOR PHASES I, M, AND HI 



Telephone Number: 



ORGANIZATION (Division, Branch, etc.): 



How long have you worked in ISD? 




Job Aids are being prepared for individuals who are performing the activities called , 
for in TRADOC Pamphlet 350-30 (Interservice Procedures for Instructional Systems Devel- 
opment), Phases I, II, -and in. These, aids will be given to USAES personnel to try out and 
evaluate in the months ahead. ; 

In January and February 1978 some USAES individuals from ITADD and CDD were 
surveyed as to the kinds of help they need in their work/ The statements presented on 
the following pages represent the outcome Qf this survey. 

* : * » **** * . t 

We^would like for you to read each statement and check whether you Agree or Dis- 
agree with it. The scale presented has four points so that yoitcan ihdicate the level of. * 
your agreement or disagreement If you cannot judge a particular statement because you ' 
have not worked in that phase of ISD, check the.box labeled "Cannot ?ud|e." Please 
check the "Cannot Judged box only when absolutely necessary. A page has also been 
provided for you to add other assistance you»need in your work. 

Thank you tot your cooperation. Your response to the statements, and the added ' 
comments, will provide greater assurance that the aids that we develop will indeed provide 
the* needed assistance to you\in your job. 



GENERAL 



1. • Wt nttd complete approved exemplas 
of the products wt art to product. 

.2? Wenetdhtlp in finding sourcts of-." 
* Utformatiorf v 

3.' Wt need a fjtntAi'britntition to the 
count development process. 



Strongly 



Strongly 



Agree * ' Agree • Disagree * Piiagree 



□ 



□ 



Cannot 
Judge 



"ISO BLOCK 1.2: SELECT. TASKS/FUNCTIONS 



I' ' 



4. Wt nttd help in identifying and 
defining standards for tasks. • 

5. Mora information is Attded on.coir- J. : m 
ditions of task performance. . 

, \ m r ; • _ r - j - 

6. We need .assistance in selecting' critical 
tasks at the appropriate skill levels. 

7. Examples of good task statements are 
* needed.' 

8. CWrly spelled <oujt proctdurts are nteded 
for identify^ an£ describing critical 
tasks and associated performance 
standard! 



ISD BLOCK 1.3: CONSTRUCT JOB 
PERFORMANCE MEASURES 

& 9. More accurate information is needed 
regarding 'the sequence of steps in task 
.performance. 

♦ » 

10. . Realistic information is needed regarding , 

actual job performance available equip- 
mtnt t end referencet * 

11. Mpre complete and detailed information is 
needed regarding task performance require- 
ments to include enabling skills/knowledges. 



□ 



<s — 

□ 




/ 

/ □ 



□ 
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Strongly 
Agree 



. ISO BLOCK 1.4: ANALYZE 
EXISTING COURSES 



12. -. Oocumantation-is natdad orfpravioui . 

training analysit and dicWoni , . * Q 



ISO BLOCK 1.5: SELECT .- 
INSTRUCTIONAL SETTING. 

13. Mori guidance b heeded qn how to select \ 
. ( from i ill possible training ewironmants. - CT 

U; We need more guidance on the purpose,; 

content and format of Soldier's Manuals.. ' □ 



ISD BLOCK 11.1: DEVELOP OBJECTIVES 

15. Mora guidanca .^needed on how to prepare 
learning objectives. □ 

* < 

16. Wa natd guidance on how to idantify and 
analyze a task's skill jnd knowladga 

components. ^ 

17. Examplas.of appropriately identified skills 

and knowledges are needed. □ 



ISD BLOCK 11.2: DEVELOP TESTS 

18. Waited. a standard format for preparing 

SQT'a. ' E 

19. We need a redesigned "Brown Book" 

■ containing exajnples of SQTi □ 

20. There is a need for aids in preparing 

course criter ion tests. • □ , 

21. We need guidance on how to develop self- 
evaluatidiTchecklistsandscorasheets. " □ 

* .22. Guidanca on how to write ggod test ques- 
tions]* needed. • □ 

23. Guidanca is needed on how to determine ** 
' 4 the difficulty of tast questions. □ 



ISO BLOCK 11.3: DESCRIBE strongly 



Strongly 



ENTITY BEHAVIOR > Agrte Agree Disagree Disagree 

I *" * 

No Aids were recommended 

'i • » 

. 

ISO BLOCK 11.4: DETERMINE 
SEQUENCE AND STRUCTURE 

24. Examples of count maps art needed. □ □ □ □ 

' 25. Examples of prescriptive outlints art needed. □ , □ □ n 

ISO BLOCK 111.1: SPECIFY LEARNING 
EVENTS/ACTIVITIE^. ~ 

26. Wanted help to identify the critical 
instructional mattriak t □ □ □ □ 

•p * — 

ISO BLOCK 1 1 1.2: SPECIFY INSTRUCTIONAL 
MANAGEMENT PLAN AND DELIVERY 
SYSTEM -v 

27. ' We nwd guidance on how to select media 

appropriate to specific objectives. , □ □ □' . □ 

28. " Guidance is needed on how to develop 

an instructional management plan. □ □ n □ 

ISD BLOCK 111.3: REVIEW/SELECT 
EXISTING MATERIALS 

! - 

29. Wt heed guidance oh how to locate, select, 
~ and review existing materials. • □ □ □ □ 

ISD BLOCK 1 1 1.4: DEVELOP INSTRUCTION 



30. Guidance is needed on the selection, use, and 



\ 



evaluation of visuals. □ □ □ ; ' □ 



31. Aids are needed for. story boarding sound- 

' slide lessons. - □ □ □ □ 

32. Guidance is needed on how to writa affectively. □ □ □ □ 

33. Aids are needed for obtaining and using 

student data to identify weaknesses in * 
i. training materials/ □ □ □ fl 

ISD BLOCK 111.5: VALIDATE INSTRUCTION 
No Aids ware recommended. / 

■ • <» 

I - ■ . 43 



PIMM indicate in tht space Wow other assistance you mid in your work-. Also. please identify, if possible, 
tht ISO Block to which this aid will bo appropriate. x 

' ASSISTANCE NEEDED * - - ' ISD BLOCK 



V) 
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APPENDIX C 

NEEDS ASSESSMENT INVENTORY FOR ISD PHASES IV AND V 
JOB AID DEVELOPMENT 



IV.1 IMPLEMENT INSTRUCTIONAL MANAGEMENT PLAN 

,, * * 

Sample: Chief (or his Representative) of Each Academic Department 



QUESTIONS 



1. 
2. 
3." 
4. 



5. 
6. 
7. 



-Name 

•*» i 

»- • _ 

if * 

Academic department 

Courses taught in department— student load and frequency of classes 
For which courses did DTD provide: _ 

a. firont-end analysis (Phase I) 

b. design products (Phase II) 

c* course development products (Phase III) • 

d. instructional management plan * * , 

e. instructor's manual 

L student's manual ' ^ 

g. POI - 

h. instructional content 

Were modifications needed in any of the products in 4 above? -If yes, who made 
these changes?' (That is, did you send them back, or do it yourself?) 

If not provided by DTD, are the products in 4 above available? Who prepared 
them? , . 

If DTD did not provide any of the products listed in 4 above:' 
Why didnt they? Organizational problems? Time, etc.? . 



What job aids are available for the implementation of the instructional management 
plan? 



a 

9. What additional job aids would be useful?* 



.ERIC, 
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° IV,2 CONDUCT INSTRUCTION - 

* * * * 

Sample: From each, academic department, as appropriate, one or two instructors 
selected on the basis of : 

1. Type of instruction presented, e.g., self-packed, CBI, conventional, etc. 

2. Instructional experience * .* 

3. Position within instructional team 

4. Student load 1 



' QUESTIONS 



1. Name 

2. Course taught x 

3. Instructional responsibility 



4: Student load 4 

a. per class . 

b. per year 

How much time is there between classes for lesson changes? 

5. Instructional experience-type, length of time, etc 

6. Subject-matter experience 

7. Who developed your instruction?^ 

8. How frequently are changes made in: ; 

poi ■ % 

b. Instructor's Manual ^ 

c. Student's Manual % ~ m m 

d. instructional content 

e. records maintained - ^ 

. t student load . • 

g. lesson plans * 

9. Who makes the changes in the materials above? 
a. Are changes fed back to DTD? ■ " * # 

10. How frequently does management plan change due to: 

a. unexpected increase/decrease in-student loan 

b. . equipment shortages 
d environment 

d. > supplies and materials 

e. availability of instructors 9 * • 



37 
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11. What forms are you required to submit-to whom?— copy available. 

12. What job aids are available. to you? (FM's) . ' 1 

13. W^at aids would be useful to you in presenting instruction? 



r 
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Y.I CONDUCT INTERNAL EVALUATION - 

* * 

Sample: Representative From Internal Evaluation Group. 



QUESTIONS 

1. Name / % 

2.. Department ^ t 

3. Is there an internal evaluation conducted for each course for which the signal school 
has propanency? If so: 

a. Who conducts it? f { 

b. When is it conducted— how frequently? * 

c. Wha^are the main steps of the evaluation? 

cL What guidance is available (TRADOC, local SOP, other (get copies). 

e. What standard forms are used? (Get copies) 1 

f. ^ What other data is collected? - 

g. How is the data summarized? • c 

h. How are the results used? Who are they sent to? 
L How do the results feed into other ISD Phases? «> 

(1) If not used what are the reasons? - ' 

(a) timeliness 

(b) they have their own source 

(c) format 

(d) other 

4, What problems exist, especially between divisions? _ " , ' 

5. How does ISD V.1 differ from IIL5? : 

6, What aids would be useful? 

o 

• <3v ■ <i 



V.2 CONDUCT EXTERNAL EVALUATION 
, V;3 REVISE SYSTEM 

* - -* 

Sample: One or more individuals from Directorate of Evaluation 



QUESTIONS 

1. Name * 



. 2, Who determines the evalJLtion procedures used? TRADOC, local SOP, other. 

3. ,If mailed questionnaires are used in the evaluation: 

/ ;1u What>1he sari&ie size? 

b; „Who is sampled? e.g. ? graduate, supervisor, etc^ 

c. How soon after graduation is individual surveyed? . 

] jl Are questionnaires MOiS specific? If yes; 

(1) Are copies available? * 1 T 

(2) What is the level of task description contained on questionnaire? 
If no, obtain copy of general questionnaire* 

' e. How is the questionnaire administered (e.g., mailed to individual, included 
in 201 file, mailed to unit, etc.)? 
£ What control procedures are used to insure questionnaire return? 
g: What is the rate 61 return? ' J 

h. Are there any follow-up procedures? 

i. Prior to graduation, what information is provided student concerning field 
feedback (i.e., answering questionnaire)? 

4. For on-site evaluation visits 

a. What MOS's sampled 

b. Frequency of evaluation 

c. Sampling procedure and size % 
d:. Make-up of evaluation team 

e. What datajs collected : 

5* How is evaluation data summarized? (mail, on-site) 
6* How useful is the data from the mailed questionnaires? 

7. Who are the results sent to? (mail and on-site) 

8. Wow <lo the results feed into other ISD Phases (i.e., do they use it)? 
If notfused, what reasons are given: 

a. Timeliness ' 5 

b. Data available from other sources 

c. Format - • 

'd; (Other * 
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& What role, if any, do SQTs play in the external evaluation?' How are results used? 



10. Of the guidance, procedural, and formi, (get copies)jised by your people, which 
* would be especially useful to others? What'problems that you hid in past were 

—solved?— — ' 

11. What additional guidance, or job aids, would you like to have for new people 
entering your directorate? Why? What problems would they solve? 



"at 



4t 
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APPENDIX D 

BACKGROUND INFORMATION 
ISD ACTIVITY CHECKLIST 



r 



Please answer each question as it pertains to your current work and/or previous experience. All responses will* 
be CONFIDENTIAL. This information will not be reported on ^n individual basis. to assure the privacy and 
anonymity of your answers. 




2. Grade 



□ 
□ 



□ 

a 



5. Branch 



7. Telephone #- ' '• 



c 

i 



ve you served in this job? 



Please check (V) in the c^art below all the ISD activities in which you have had some experience . 



^Preparing critical tat KJiSting 
* Conducting field validation of critical task list 
Preparing job task performance measures 
Prepering.tesk summary sheets/job task data cards* etc. 
Preparing list oUreference materials for CM/SM | 4 

□ Selecting, instructional set ti nowhere training is to be conducted) 
- D t Writing training objectives (HL&s, LOs, CLOs, and t$s) 

□ { Preparing test administration. rnao.ua I <TAM) 

□ j Preparing SQT materials 

□ ^Preparing other tests (criterion* ( 

□ [Conduct ing validation of tests by subject matter experts 

□ Conducting field validation ot tests 



□ Making media methods selections • 

□ Locating selecting, and reviewing existing materials 

□ Oevelopmg correspondence sgbcourse a . • 

□ Oevelopmg TEC^materials"^ 
DeveloprngTesidentHesson plan > x 

~~T5 Developing P.E.. sheafs (workbooks) 

□ Developing self-paced performance (lesson) guide 

□ Developing AV scripts, T.V. scripts, etc. 

□ Developing programmed text, CAl, etc. 

□ Developing other instruction (resident/non-resident) * 
- □ Conducting developmental testing (one-on-one) ' 

□ Conducting small group validation trials 
* D Making revisions following small group validation trials 

□ Conducting large group. validation trials 

□ Other (specify) 



■ □ Making revision following validation tes 
1 G examining student entry behavior 
, D Sequencing of instructional material (course map) 
;'□ Classjfyingrtearning objectives (elements) by^ypes of learning 

12. Please check (V) in the chart below, al| the ISD activities that you have performed or will perform on a 
single MPS. 



□ Preparing critical task listing 

□ Conducting field validation of critical task list . 

□ ^Preparing Job task performance measures 

□ Preparing task summary sheets/job task* data cards, etc. 
O Preparing Hit of reference materials for CM/SM 

' □ Selecting instructional setting (where training is to be conducted) 

□ Writing training objectives (TLOs, LOs, CLOs, and LSs) 

□ Preparing test administration manual (17AM) ' 

□ Preparing $QT materials # j * 

□ Preparing other tests (criterion; etc.) x 
'□ Conducting validation of tests by subject matter experts 

□ Conducting field validation of tests 0 

□ Making revision following validation tests 
>0 Examining student entry behavior . 

□ Sequencing Of Instructional material (course map) 

□ Classifying learning objectives (elements) by types of learning 



ERIC- 




D Making media methods selections 

G Locating selecting, and reviewing existing materials 

□. Developing correspondence subcourse 

□ Developing TEC^materials 

□ Developing resident lesson plan ' ', 

□ Developing P.E. sheets (workbooks) 

□ Developing self-paced performance (lesson) guide 

□ Developing AV scripts, T>V. scripts, etc. ' . i } 

□ Developing programmed text, CAl, etc. 

□ Developing other instruction (resident/non-resident) 

□ Conducting developmental testing (one-on-one) . 

□ Conducting srpell group validation trials 

□ Making revisions following .smell group validation trials 

□ Conducting large group validation trials 
Other (specify) ' ; t 
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ISD ACXUtf TY CHECKLIST 



Mease answer «ach question as it pertains to 4 the Instructional Systems Development (ISD) activity you hdve worked 
on during thtf past Jwo weeks. This information, will not be reported on an individual basis to assure .the privacy* £ 
arfcT anonymity of your answers. If you have any additional comment?* please enter them at the bottom of the form: 

1. Name 2. Oate 



3. Please check </) in 'the chart below the ISD activity that you have worked on during tfie past £ weeks. 



□ Prepering critical task lilting 

□ Conducting field validation of critical task list - 
U Preparing job task parformanca meesures ' 

□ Prepering task summer shee ti/job task data cards/etc. 

□ Preparing tilt of raf aranor materials for CArf/SM - 

□ Salacting instructional sitting (where training is to be conducted) 

□ Writing training objectives (TCOs, LOs, CLOs, and ISs) 

□ Preparing test administration manual (TAM) 

□ Preparing SOT metefials 

□ Preparing othar tests (criterion, a\c.) . * , 

□ Conducting validation of tasts by subject mejter experts 

□ Conducting field validation ot tasts 

□ Making revision following velidetion tests 

□ Examining student entry behavior 

□ Sequencing of instructional material: (course map? . 

□ Classifying learning objectives (elements) by types of learning 

4. bid you complete (at least in draft form), the activity checked above? Yes.. , , No. _ 

5. Estimate the tQtal number of hours that you spent working on the ISD activity checked off above. 

6. What percentag^\of the time that you worked on this ISD activity was spent om 

(a) Initial Preparation » 

(b) Review/Revision % * . 
' <c)* Other (explain) - % 



□ Makipg media methods selections 

□ Locating selecting, and reviewing existing materials 

□ 'Developing correspondence subcourse # 

□ Developing TEC materiets ^ . \ 

□ Developing resident lesson plan . . \ f K 

□ Developing P.E. Sheets (workbooks) 

□ developing self-paced performance (lesson) guide 

□ 'Developing A V scripts; T.V. scripts^etc. 
* n>Developir»j /progremmed text, CAt, etc. • ° 

□ Developing other instruction (resident/non-resident) 

□ Conducting developmental testing (one*on-one) 

□ Conducting s/nell group validation trials / * 

□ Making revisions following smalt group validation trials 
. □ Conducting targe group validation trials 

□ Other (specify) 



. Hours* 



7. What problems (if any) did you experience in performing your work in this activity? (Please check all that 
applicable.) . ^ ^ ' [ 



are 



(a) Guidance was not available * 

(b) Guidance was inaccurate 

(c) Cogld not get needed information 

(d) Information was inaccurate 

(e) References were out of date 

(f) Did not receive enough supervision 

^q) » Not enough help from other personnel , 

(nA 

Other (please describe) - 



Had no problems < 



Input te my fSD work.yvas unsatisfactory 
Input to my ISD work wasl^te ; . 
Not enough time to gather information 

• Not enough time to develop material 

♦ 

Not enough time to review/revise material 



(h) 
(i)' 
(i) 
(k) 

(I) 

(m) I did not receive enough training to do this 
work satisfactorily 

c 



8* If the input to your ISO work was not satisfactory", What dhj you do? - 

(a) Reported it to my supervisor 'j. 

(b) Returned it for revision " 5 

(c) Did it over myself ' - 

, (d) Not applicable "> « 

(e) • Other (please describe) ; 
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9.?Was the output (product] fro m^your work returned to you for revision? Yes. 
If "Yes/' what had to be changed? 

(a) Errors had to be corrected : - 

(b) Format was wrong _ ! * 

(c) References were wrong . _ 

(d) Incorrect wording 4 

(e) Documentation was inadequate ' . 

(f ) Other (please describe) ' m . 



No. 




COMMENTS: . 



" ** ■ * 


\ 






J 








f 



Pfaa* piac^ this; form jn the attached envelope and place it in the HumRRO container located in 
Mr. John D. Danitovich's office (Bldg. 19704) or in Mr. John R. Myers' office (Bldg. 25701): ' 

/ Thank you for ypur cooperation. 



*. { 
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Interviewer 



Date 



APPENDIX E 
LOW-W 



JOB AID FOLLOW-UP INTERVIEW FOl 



' BACKGROUND INFORMATION 

- a- ■ 



Name 



Grade * 



Divisioh/CMF V 



MOS (working on) 



• 1 



Current Job 



» . . Ho/ long in job? 



- / 



! How long in ISD block for whiah Job Aid designed? 

• ■ • ' S - . " ■ ■ \ . * ' ' ' s 

>Any formal training iri this Ics'ty block? If yes, ^hat and how much? 



\< 



Si 



\ 



Did you complete a Background. Information Sheet a* few months ago' 
^<theA complete /a Base-line Dat# Background Information Sh6et. 

Who is. your supervisor? 



If no$ have 



FLOWCHART, 



1.. To what extent did you actually use the Flowcharts? 



Were the symbols used underf tandable? 



3. Was written materia^ on side understandable and useful? 



f 

l- 
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; FLOWCHART (Continued) 



A... Here there times when you were referred to the manual unnecessarily? Times when you 
- should have been referred and were not? " Which blocks? 



ml 5/ Did you ever go to" the Manual to see an example when not required to do so? If yes, 
- how frequently? ... ^ 



^ 6# Jfas there a need for expansion/collapse of some flowchart blocks? If yes, where? 



; 7. Were there too few/too many supervisor checks of your work called for. in the aid? 
Where J should they have been added or deleted? 



8.* What is your overall evaluation of the, flowcharts? How can they be improved? 



4 * 



^: >IHIMPDCnOlt TO JOB- AIDS 

. . <i , 

<V9# -How food a job did it do in explaining what the Job Aids were? How can it be 
; \ improved? 




FT* 



■t - . 



r- 



•0 



'■*/.•* 1 ■ 

' /; RESOURCE MANUAL ' , n 

j/ * 10. T)id you use the Resource Manual? How yell^id it cover: CODAP, School Conducted 
Field Surveys > and School convened panels? 

t 



I! MANUAL 

' *« 7 — * 

'V 



- i-,"7 

7 

•J- 

* i 
V 

X \ 



\ 11. Were the partial flowcharts at the top of jthe page useful? To what extent did 



you 



"use them? 



. ,12» * Was the overview to the job aid useful to you? How can it be improved? 

',1 • • - \ 

l^. Were the- questions used to introduce each area useful? 



14.* Is the text in the "manual tqo difficult "to read? Explain. 



15. Were there inaccuracies in the information presented? Explain. 

/ 

/ 
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MANUAL (Continued) 



16. Were Important sources of information omitted? If yes, what are these sources? 

. - • f ■ 

[ 



17 • Was inappropriate or obsolete information presented? Explain • 



ia. How useful were the references provided? Were they correct? What additional 
references or sources of information do you usually use? 



19. How useful are the examples provided? 



\ 

\ 



/ 



/ 

i 
i 



20 



'What is your overall evaluation of the Manual? How can itxfce impro^gd? 



\ 



x 



WORKSHEETS 



21. How useful were the Worksheets provided? How can they be improved? 
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WORKSHEETS (Continued)' ' , 

22. J)id*the Worksheets require t6o little/too much documentation? 



GENERAL 



23. Was the. reading level of the Job Aid (flowcharts, introduction, etc.) too high/ 
too low? * * 



24. Wtu*t did you do before you had this Job Aid that was better or more efficient 
for performing the work in this area? 



25* Do you, think that by using this Job Aid you have turned out a better quality 
product than without the aid? 



26. In the long run, do you think that use of the Job Aid will turn out products 
faster or slower than without the aid? 



27/ Vo you think the Job Aid is too long/too short or about right in length? 

49 
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GENERAL (Continued) < 

28. Will you continue to use this aid in the future? If no,^find out why. 



29. What type of person do' you think would find this aid useful? (i.e., inexperienced/ 
experienced) 



30* In order to use' the Job Aid you needed materials from individuals who had 
performed ISD activities preceding yours. Did the materials used hinder 
or help you in your work? 
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1 USA iRANSPORTAUON SCHOOL OEPUl Y ASST. COMMANDANT EDUCA. TECHNOLOGY 
1 USA SIGNAL -SCrtDOL AND FT. GORUON • ATTN: ATZH-ET 
TlJSA viUARTERMASTER SCH ATlN: ATSM-1 NG-TM-ET ... 
1 USA MILITARY POLICE- SChOOL AT TNr LIBRARY 

1 USa.aRMOR CENT5.R FT. KNOX OFFICE OF' ARMOR . FORCE MGT ♦ STANDARD^ HON 
-1 CHIEl- OF NAVAL ED JC AT I ON cAND J»G / - » 

1 USA alGNAL SCHOOL v FT. GORDON EDUCATIONAL TECHNOLOGY DIVISION 
-_.*] HO.AIC/XPTO TRAINING SYSJ.tMS DEVELOPMENT 

1 USA INSTITUTE FOR MILITARY ASSISTANCE ATTN: ATSU-TO-TA ' 

■'\- t us cAkmy armor center attn: atzk-td-hmo 

1 USA mIR OEFEN> b CENTtR* F I .' BL JSS ATTN: aTZC-DIM \ 
1 USA uUARTERMAS»rR % SCHODL~~D l RLC.l ORATE OF 1 RAININ G— DEVELOPMENTS 

i us cuast guard acadlmy /' ' 

1 USA IRANSPORTAT 10 V. SCHOOL DIRECTORATE OF TRAINING ♦ DOCTRINE 

l.USA iNFANTRf SCHOOL LIbRAKY„ / 1 . ' --- 

~ I'USA iNFANTRY SChODL' ATTN? ATSh-1-V 
t US AhMY 'INFANTRY SCHOOL ATTN: ATSH-CO 
1 USA INFANTRY SCHOOL ATTN: ATSh-UOI-LRU 

1 USAOiNFANTRY SChODL ATTN: ATSn~tV ' ' 
,1 USA rtP ♦ CHEM SCH/TNG CtN ♦ FT. . MCCLELLAN ATTN: ATZN-PTS \ 
I USA «P ♦ CHEM SCH/TNG CEN ♦ FT. MCCLElLAN DIR. COMBAT DEVELOPMENT 
.1 US« rtP ♦ CHEM. SCH/TNG" CEKi ♦ FT. MCCLELLAN UIRj TRAINING DEVELOPMENT 
1 USA i«iP ♦ CHEM SCH/TNtj CEN ♦ FT. MCCLELLAN ATTN: ATZN-MP-ACE 
I USA INSTITUTE OF ADMINISTRATION ATTN: RESIDENT TRAINING MANAGEMENT 
. 1 USA 1 1ELD ARTILLERY SCHOOL MORRIS SwEM LIBRARY 
1 USA iNSTITLTE OF ADMINISTRATION ACADEMIC LIBRARY 
1 USA wAR COLLEGE ATTN: LlBHARY 

•1 US« tNGlNEER SCHODt LlBRAHY* AnD LEARNING RESOURCES CENTER 
' 1 USA' mRMOR SCHOOL (USARmS) ATTN: LIBRARY 
1 US COAST GUARD ACADEMY LIBRARY 

1 USA iRANSPORTATlON SCHOOL TRANSPORTATION SCHOOL LIBRARY 
1 ORGANIZATIONAL F.F PECTI VENESS CF.N * SCH ATTN: LIBRARIAN 
1 US AkMY INTELLIGENCE CENTfcK ♦ SCHOOL ATTN: ATSI-TP - . 

1 US AKMY INTELLIGENCE CENTER ♦ SCHOOL ATTN: ATSI-RM-M 
1 US AKMY INTELLIGENCE CENTtR ♦ SCHOOL ATTN: ATSI-ES 
I US MmHINE CORPS EDUCATION CENTtR 

i USA t IELO ARTILLERY SCHOOL DlRtCTORATt OF COURSE OEV ♦ TRAINING 
Y^i «)EPAhTMENT Of I HE AIR FORCE AIR UNlVcHSlTY LIBRARY (ATC) , 
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I USA CHAPLAIN CtNTEH ♦ SCHOOL ATTN: ATSC-TD-00 

t USA CHAPLAIN CtNTEH bCHOOL ATlNt ATSc-TD-ED 

1 USA. CHAPLAIN CLulEH*'*- SCHOUL AlTN: AlSC-TD-SF - ----- 

1 USA CHAPLAIN 'CtKlt EH ♦ . SCHOOL ATINj__AlSG-t)OS--LLC 
. 1 HQ SkADQC .TRAINING DEVELOPMENT INSTIIUTf 

2 ftftlTiSH EMBASSY HRITISM OtFENCt STAFF- 
? *CANAulAN "JOINT STAFF 

* 1 COLS (W) LIBRARY ' " 

- 1 FRENCH ARMY ATlACHt 

1 AUSTRIAN EMBASSY UFFENSE* MlLHAJRY A MO AIR ATTACHE 

3 CANAuIAN DFFENCt LU1SON SlAFF ATTN: COUNSELLOR t OEF'ENCE R AND 0 
' I HOYAL NETHERLANDS fcMBASSY -Ml L1 1 ARY °AT lACME 

1 CAMAolAN FORCES' BASE CORNWALLIS ATTN: PERSONNEL SELECTION 

? CANAoIANi FORCED PERSONNEL APPL.RSCh UNIT .. 

1 AR-'^Y PERSONNEL RtStAHCH ESTABLISHMENT ( . 

°6 Ll'tiriMRY- OF CONbRtSS EXCHANGE A NO GIFT 01 V fc * « ' 

1 DEFENSE TECHNICAL INF,0RMA I ION CEN AT In» uTlC-DDA-,2 

140 LIBRARY OF CONFESS UNIT DOCUMENTS EXPEDITING PROJECT 

1 US GO.VEHNMENT Pm1>JTING OFC LIhHAHYi PUBLIC DOCUMENTS DEPARTMENT 

i us government phi>jting ofc % libhary "and statutory, lib div (sld 

I THE aRMY LlBRAKf ATTN: ARMY STUDIES SEC ' « 

? / / 
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